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ABSTRACT 


The  authors  have  developed  a  prototype 
computerized  model  for  acquisition  strategy  compa¬ 
rison.  An  interactive  menu  selection  process  is 
used  to  obtain  a  general  description  of  the  weap¬ 
on  system  concept  and  program  objectives.  The  mod¬ 
el  and  the  user  then  interact  to  successively  re¬ 
duce  the  number  of  strategy  alternatives  to  a  small 
set  containing  the  preferred  alternatives  for  a 
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EXECUTIVE  SUMMARY 


Under  the  sponsorship  of  the  Defense  Systems  Manage¬ 
ment  College  (DSMC) ,  The  Analytic  Sciences  Corporation  (TASC) 
has  developed  a  prototype  Acquisition  Strategy  Compari son  Model 
( ASCM ) .  The  objective  of  this  phase  was  tc  develop  and  demon¬ 
strate  a  computerized  model  and  data  base  for  evaluating  acqui¬ 
sition  strategy  alternatives.  Results  demonstrated  both  the 
feasibility  of  such  a  concept  and  general  agreement  with  pro - 
gram  experience . 

The  scope  of  the  current  effort  was  intentionally 
limited  to  two  categories  of  weapon  systems,  tactical  missiles 
and  selected  electronic  subsystems.  The  data  base  developed 
consists  of  data- on  37  programs  plus  ancillary  data.  It  is 
adequate  for  prototype  demonstration,  but  cannot  be  considered 
complete  for  full  prog ran  application.  Data  collection  and 
analysis  efforts  for  both  the  current  effort  and  potential, 
future  phases  are  documented  herein. 

The  model  is  structured  to  duplicate  a  logical  pro¬ 
cess  which  might  be  employed  by  a  program  manager  to  select  an 
acquisition  strategy  for  his  program.  This  process  is  summa¬ 
rized  as  follows: 


•  Identify  feasible  strategies 

0  Reduce  the  set  of  feasible  strategies  to 
a  set  consisting  of  those  strategies 
with  the  highest  probability  of  achieving 
the  desired  result 

0  Further  reduce  the  set  of  possible  strat¬ 
egies  by  funding  limitations 
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©  Narrow  i he  choices  even  further  by  elim¬ 
inating  "second-best"  options  via  detailed 
comparative  analysis. 

This  successive  reduction  process  typically  results  in  delet-* 
ing  a  vast  majority  of  the  possible  strategies,  leaving  a  small 
number  of  strategies  for  further  in-depth  tradeoff  analysis. 

The  model  also  presents  the  relative  costs  and  benefits  asso¬ 
ciated  with  the  different  options,  and  it  can  be  exercised  to 
highlight  sensitivities  to  program  or  risk  variations. 

The  report  provides  a  detailed  description  of  the  en¬ 
tire  process,  an  illustrative  example  incorporating  sensitiv¬ 
ity  analysis,  and  a  real-world  example  of  results  from  the 
AN/SLQ-32  program  (a  Navy  ship-based  electronic  warfare  .program) . 

These  examples  provide  ample  evidence  that  the  proto¬ 
type  model  provides  results  consistent  with  program  experience. 
Furthermore,  they  demonstrate  the  insights  the  model  is  capable 
of  providing: 


•  To  illustrate  the  interrelationships 
among  acquisition  strategy  alternatives 
and  key  influencing  factors 

•  To  emphasize  that  acquisition  strategy 
encompasses  the  entire  acquisition 
process  and  demonstrate  chat  the  selec¬ 
tion  of  an  acquisition  alternative  for 
one  phase  should  not  be  made  indepen¬ 
dently  of  other  phase  options 

•  To  highlight  the  importance  of  risk 
identification  and  risk  management  early 
in  the  program. 

Collectively,  these  findings  provide  a  strong  indication  of 
the  potential  utility  of  the  model  as  a  teaching  aid  to  pro 
gram  management  students  at  DSMC  and  elsewhere . 
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Combined  with  more  in-depth  data  collection  and  anal¬ 
ysis  to  support  the  myriad  of  internal  relationships,  ASCM 
could  also  pr-ovide  early  planning  support  to  program  manage¬ 
ment  personnel. 


Finally,  the  data  base  and  analytic  structure  provide 
an  excellent  first  step  toward  the  development  of  a  unique  and 
badly  needed  acquisition  strategy  research  tool. 


Based  on  the  success  of  the  prototype  model,  TASC 
strongly  endorses  the  follov?ing  recommendations: 


o  Implement  ASCM  as  a  teaching  aid  in  the 
program  management  curriculum  at  DSMC 
(through  the  development  of  user- 
friendly  software) 

•  Perform  additional  data  collection  and 
analysis  of  tactical  missile  system  and 
electronic  subsystem  development  and 
production  phases  in  order  to  validate 
and  refine  key  model  parameters  and 
relationships.  (This  will  vastly 
increase  the  validity  and  realism  of  the 
prototype  model.) 

•  Perform  additional  research  and  analysis 
to  refine,  update,  modify,  and  validate 
(as  required)  two  key  methodologies 
incorporated  in  the  model  (again,  this 
will  vastly  increase  the  validity  and 
realism  of  the  prototype  model) 

•  Expand  the  scope  of  the  model  to  include 
additional  categories  of  weapon  systems 
in  order  to  provide  maximum  utility  to 
the  overall  defense  community 

a  Perform  research  into  the  feasibility  of 
expanding  the  scope  of  the  model  to  in¬ 
clude  a  model  of  the  concept  exploration 
process.  (Given  such  a  model,  a  user 
could  begin  with  a  perceived  operational 
need,  determine  likely  concepts  to 
address  that  need,  and  then  assess 
tradeoffs  among  the  competing  concepts.) 
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1. 


INTRODUCTION 


1 . 1  BACKGROUND 


Under  the  sponsorship  of  the  Defense  Systems  Manage¬ 
ment  College  (DSMC),  The  Analytic  Sciences  Corporation  (TASC) 
previously  assessed  the  feasibility  of  developing  an  analytic 
model  to  aid  in  selecting  an  acquisition  strategy  for  research, 
development,  and  production  of  major  weapon  systems.  Results 
from  that  assessment  of  significance  to  the  current  effort  are 
as  follows: 


•  The  elements  of  acquisition  strategy 
were  described  as  a  set  of  strategy 
alternatives  over  four  acquisition 
phases 

•  Preliminary  estimates  of  quantitative 
relationships  were  developed  which 
indicate  the  expected  result  of  pursuing 
a  particular  acquisition  strategy 

•  Examination  of  four  major  weapon  system 
acquisition  programs  indicated  a  suffi¬ 
cient  quantity  and  quality  of  data  to 
support  the  development  of  an  historical 
data  base,  but  an  intense  effort  was  re¬ 
quired  for  collection 

•  The  development  of  an  analytic  model  for 
use  in  selecting  an  acquisition  strategy 
was  deemed  feasible.  The  quantifiable 
relationships  developed  would  measure 
the  expected  result  jf  pursuing  each 
feasible  acquisition  strategy.  To 
evaluate  the  strategies,  TASC  recom¬ 
mended  the  use  of  decision  analysis 
techniques  and  the  development  of  a 
multi-attribute  utility  model  which 
could  be  tailored  to  meet  the  needs  and 
constraints  of  each  particular  program 
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•  A  three-phase  approach  for  full  model 
development  was  recommended.  Phase  I 
consisted  of  the  development  of  a  pre¬ 
liminary  computerized  model  combined 
with  a  data  collection  effort  sufficient 
to  obtain  first-order  estimates  of 
required  parameters.  Phase  11  consisted 
of  model  evaluation  by  using  the  pre¬ 
liminary  model  to  assist  program  mana¬ 
gers  with  acquisition  strategy  decisions, 
performed  concurrently  with  additional 
data  base  development.  The  final  phase 
consisted  of  updating  the  model  to  include 
lessons  learned  during  the  evaluation 
phase,  followed  by  full  model  implementa¬ 
tion  and  documentation. 


A  complete  presentation  of  these  findings  is  contained  in  TASC 
report  TR-1375,  "Feasibility  and  Development  Study  for  a  Sys¬ 
tem  Acquisition  Strategy  Model  -  Final  Report",  dated  12  Jan¬ 
uary  1981. 


1.2  OBJECTIVE 

Not  surprisingly,  the  reactions  to  the  findings  and 
conclusions  contained  in  the  feasibility  assessment  constitut¬ 
ed  a  mixed  review.  There  was  little  disagreement  concerning 
the  basic  findings  and  even  only  minor  doubt  concerning  the 
feasibility,  in  a  theoretical  sense,  of  developing  a  model 
along  the  lines  described.  The  main  concern  was  one  of  imple¬ 
mentation.  Was  it  possible,  in  a  real-world  sense,  to  develop 
and  implement  such  a  model  that  would  provide  reasonable  re¬ 
sults?  In  an  attempt  to  reduce  the  uncertainty  associated 
with  realistically  implementing  such  a  model,  the  scope  of  the 
subsequent  phase  was  altered. 
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The  main  objective  for  this  phase  was  to  develop  and 
demonstrate  a  computerized  model  and  data  base  for  evaluating 
acquisition  strategy  alternatives.  Specific  tasks  were  as 
follows : 


•  TASK  A  -  Demonstrate  an  acquisition  strat¬ 
egy  model  for  two  categories  of  weapon 
systems  or  subsystems.  The  demonstra¬ 
tion  shall  be  structured  to  illustrate 
the  complex  interrelationships  for  all 
significant  program  variables  discussed 

in  Phase  I,  to  form  the  basis  for  approval 
of  Phase  111,  and  to  provide  an  early 
indication  of  implementation  requirements, 
i.e.,  input-output  data,  computer  capa¬ 
city,  facilities,  and  the  like.  Compari¬ 
son  with  program  experience  shall  be  the 
basis  for  determining  adequacy  of  the 
demonstration 

•  TASK  B  -  Provide  DSMC  with  two  copies  of 
all  programs,  subroutines,  modules  and 
access  to  data  associated  with  TASK  A. 
Support  DSMC  analysis  of  program  data, 
algorithms,  or  output  results 

•  TASK  C  -  Develop  specifications  and  re¬ 
quirements  for  Phase  111  and  Phase  IV  to 
include  time  and  cost  considerations, 
architecture,  and  interface  requirements 

•  TASK  D  -  Provide  periodic  review  meet¬ 
ings  to  be  held  at  the  completion  of 
significant  task  accomplishments. 


The  two  categories  of  weapon  systems  and  subsystems 
agreed  upon  for  the  demonstration  model  were  tactical  missiles 
and  key  military  electronic  subsystems.  The  primary  rationale 
for  their  selection  is  as  follows: 


•  Both  categories  are  used  by  all  three 

services,  thus  providing  a  broader  base 
for  the  demonstration 
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•  Electronic  subsystems,  such  as  the  guid¬ 
ance  and  control  system,  are  typically 
key  components  of  tactical  missiles.  It 
was  believed  that  interdependencies  might 
exist  which  could  be  exploited,  partic¬ 
ularly  in  the  data  collection  and  analy¬ 
sis  arena 

•  From  previous  research,  TASC  had  obtained 
production  cost  histories  on  a  number  of 
tactical  missiles  and  electronic  subsys¬ 
tems.  The  use  of  this  existing  data 
would  reduce  the  data  collection  effort. 


Two  principal  deliverables  were  associated  with  this 
effort.  The  first  and  most  important  was  a  formal  demonstra¬ 
tion  of  the  model.  This  demonstration  was  conducted  at  DSMC 
on  February  19,  1982.  In  addition,  several  members  of  the 
DSMC  staff  experienced  interactive  use  of  the  model  on  March  2 
at  TASC's  facilities  in  Arlington,  Virginia.  This  report  is 
the  other  principal  deliverable  and  it  documents  the  research 
effort,  together  with  results,  findings,  conclusions,  and  recom 
mendations . 


1.3  CONTENT  OF  REPORT 

This  report  consists  of  two  volumes.  This  volume 
contains  an  executive  summary  and  the  main  report.  The  re¬ 
maining  chapters  of  this  volume  provide  a  description  of  the 
model,  together  with  some  examples,  followed  by  the  conclu¬ 
sions  and  recommendations  resulting  from  this  effort.  Volume 
II  consists  of  appendices  which  provide  a  detailed  description 
of  the  inner  workings  of  the  model  plus  details  of  the  data 
collection  and  analysis  efforts. 


1-4 


THE  ANALYTIC  SCIENCES  CORPORATION 


2. 


MODEL  DESCRIPTION 


2.1  PHILOSOPHY 

One  hypothesis  underlying  the  model's  overall  approach 
concerns  the  effort  required  to  transform  a  weapon  system  con¬ 
cept  into  a  producible  hardware  (and  software)  configuration. 
This  effort  is  considered  to  be  relatively  independent  of  time. 
The  same  basic  tasks  are  required  to  develop  a  weapon  system 
today  as  were  required  twenty  or  thirty  years  ago.  The  order 
in  which  the  tasks  are  performed  may  differ,  the  nature  of 
overlap  among  the  tasks  may  vary,  and  the  names  associated 
with  the  activities  may  change  on  a  regular  basis.  However, 
the  basic  tasks  remain  relatively  static.  Histories  of 
missile  development  programs  support  this  hypothesis. 

ASCM  builds  upon  this  hypothesis.  Its  philosophy  is 
that  the  lessons  learned  in  prior  programs  can  provide  inval¬ 
uable  guidance  in  choosing  the  acquisition  strategy  consistent 
with  the  goals  and  objectives  of  a  program  today.  In  this 
light,  ASCM  can  be  viewed  as  a  computerized  lessons-learned 
algorithm. 

An  equally  important  philosophy  concerns  the  model's 
implementation.  There  was  a  concentrated  effort  throughout 
the  development  to  align  the  logical  structure  of  the  model 
with  a  possible  logical  process  employed  by  a  program  manager 
in  selecting  an  acquisition  strategy  for  his  program.  That 
process  can  be  summarized  into  four  key  steps: 

•  Identify  all  feasible  strategies 

•  Reduce  the  set  of  feasible  strategies  to 
a  smaller  set  consisting  of  those  strat¬ 
egies  with  the  highest  probabilities  of 
achieving  the  desired  result 
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•  Further  reduce  the  set  of  possible  stra¬ 
tegies  by  financial  limitations 

•  Perform  detailed  comparative  analysis  of 
those  remaining  to  eliminate  "second 
best"  options. 

ASCM  follows  this  logical  process. 

In  developing  an  initial  model  of  any  large  and  com¬ 
plex  process,  there  is  frequently  an  inverse  relationship 
between  the  level  of  detail  addressed  by  the  model  and  the 
probability  of  creating  a  successful  model.  The  generally 
accepted  approach  in  these  cases  is  to  keep  the  initial  model 
as  simple  as  possible  and  address  only  those  factors  deemed 
most  crucial  to  the  process.  For  our  case,  over  150  factors 
were  identified  during  the  feasibility  study  as  potentially 
relevant  to  an  acquisition  strategy  decision.  For  initial 
model  development,  the  factors  considered  to  be  most  critical 
in  the  majority  of  situations  were  combined  into  higher  level 
aggregate  factors,  and  other  less  critical  factors  were  not 
considered.  Expansion  and  refinement  of  the  model  to  include 
missing  factors,  such  as  life  cycle  cost  implications,  and  to 
expand  upon  aggregrate  factors  is  best  accomplished  after  the 
initial  model  is  demonstrated  to  be  reasonable. 


2.2  ACQUISITION  STRATEGY  ALTERNATIVES 

The  first  step  in  using  ASCM  is  to  understand  the 
concept  of  acquisition  strategy  employed.  As  background  to 
this  concept,  DODI  5000.2,  19  March  1980,  defines  acquisition 
strategy  as  follows  (emphasis  added  by  authors): 
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"Acquisition  strategy  is  the  conceptual  basis 
of  the  overall  plan  that  a  program  manager  fol¬ 
lows  in  program  execution.  It  reflects  the 
management  concepts  that  shall  be  used  in  di¬ 
recting  and  controlling  all  elements  of  the 
acquisition  in  response  to  specific  goals  and 
objectives  of  the  program  and  in  ensuring 
that  the  system  being  acquired  satisfies  the 
approved  mission  need.  Acquisition  strategy  en¬ 
compasses  the  entire  acquisition  process. 

The  strategy  shall  be  developed  in  sufficient 
detail,  at  the  time  of  issuing  the  solicita¬ 
tions,  to  permit  competitive  exploration  of 
alternative  system  design  concepts  in  the 
Concept  Development  phase.  Additionally, 
sufficient  planning  must  be  accomplished  for 
succeeding  program  phases,  including  produc¬ 
tion,  for  those  considerations  that  may  have 
a  direct  influence  on  competition  and  design 
efforts  by  contractors'!  The  acquisition  stra¬ 
tegy  shall  evolve  through  an  iterative  process 
and  become  increasingly  definitive  in  describ¬ 
ing  the  interrelationship  of  the  management, 
technical,  business,  resource,  force  structure, 
support,  testing  and  other  aspects  of  the 
program. " 


Consistent  with  this  definition,  ASCM  looks  at  acqui¬ 
sition  strategy  in  a  macro-perspective  and  considers  it  to  con 
sist  of  25  strategy  alternatives  spanning  four  acquisition 
phases  (Figure  2.2-1).  These  alternatives  encompass  the  feasi 
ble  options  identified  for  the  development  of  a  new  weapon 
system  and  for  a  modification  program  for  an  existing  weapon 
system.  An  in-depth  discussion  of  each  alternative  is  con¬ 
tained  in  the  final  report  for  the  prior  feasibility  assess¬ 
ment  (TASC  Report  TR-1375).  A  brief  description  of  the  alter¬ 
natives  by  acquisition  phase  appears  in  the  following  sections 
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PHASE  0: 

Concept  Exploration 

• 

Directed  concept 

• 

By  non-industrial  firms 

• 

By  industrial  firms 

• 

Jointly 

PHASE  1: 

Demonstration  and  Validation  (D&V) 

• 

Waive 

• 

Contract  definition 

-  by  non-industrial  firm(s) 

-  by  single  industrial  firm 

-  by  multiple  industrial  firms 

• 

Subsystem/component  development 

-  by  non-industrial  firm(s) 

-  by  single  industrial  firm 

-  by  multiple  industrial  firms 

• 

System  prototype 

-  by  non-industrial  firm(s) 

-  by  single  industrial  firm 

-  by  multiple  industrial  firms 

PHASE  2: 

Full-Scale  Development  (FSD) 

• 

Incremental  development 

-  by  single  source 

-  by  multiple  sources 

• 

Partial  concurrency 

-  by  single  source 

-  by  multiple  sources 

• 

Full  (extreme)  concurrency 
(Single  source) 

PHASE  3: 

Production  and  Deployment 

• 

Single  source,  no  options 

• 

Single  source  with  options 

• 

Single  source,  multi-year  contract 

• 

Leader- follower 

• 

Licensing 

• 

Second  sourcing 

Figure  2.2-1  Acquisition  Strategy  Alternatives 
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2.2.1  Concept  Exploration 

The  primary  objective  of  concept  exploration  is  to 
examine  feasible  solutions  of  a  perceived  operational  need  and 
to  select  for  further  development  those  solutions  which  exhi¬ 
bit  the  highest  potential.  The  alternatives  for  this  phase 
basically  concern  where  this  effort  is  performed.  Although 
time  and  cost  distributions  for  this  phase  are  included  in  the 
model,  they  contribute  very  little  to  the  insight  provided  by 
the  model.  Since  the  prototype  model  was  restricted  to  tac¬ 
tical  missiles  and  key  electronic  subsystems,  the  model  in  its 
current  state  principally  concerns  the  development  options  for 
a  pre-defined  concept.  Further  research  into  feasible 
approaches  to  modeling  the  concept  exploration  process  and  its 
function  would  be  appropriate. 

2.2.2  Demonstration  and  Validation 

The  principal  objective  of  the  Demonstration  and 
Validation  Phase  is  to  demonstrate  that  the  technology  re¬ 
quired  to  implement  a  particular  concept  is  primarily  an 
engineering  application,  rather  than  an  experimental  effort. 
The  alternatives  for  this  phase  span  two  dimensions.  The 
first  dimension  concerns  the  scope  of  work  performed.  The 
four  options  in  this  dimension  range  from  doing  nothing 
(Waive),  to  paper  analysis  (Contract  Definition),  to  building 
select  hardware  (Subsystem/  Component  Development),  to  build¬ 
ing  a  system  prototype.  The  second  dimension  concerns  the 
organization  performing  the  work. 
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These  options  consist  of  non-industrial  firms*,  a  single  indus¬ 
trial  firm,  or  multiple  industrial  firms. 

2.2.3  Full-Scale  Development 

The  primary  objective  of  Full-Scale  Development  is 
intense  analysis  and  refinement  of  the  system  design 
(including  peripheral  equipment)  to  ensure  that  the  pre- 
production  prototypes  will  meet  performance  thresholds.  The 
alternatives  for  this  phase  principally  concern  timing,  or  the 
degree  of  concurrency  employed,  and  the  number  of  development 
firms . 

2.2.4  Production  and  Deployment 

For  Phase  3,  Production  and  Deployment,  the  options 
concentrate  on  the  single  source  versus  dual  source  decision 
with  variations  allowed  for  each  case. 

An  in-depth  discussion  of  each  alternative  is  con¬ 
tained  in  the  prior  feasibility  study. 


*The  term  non-industrial  organizations  is  used  for  brevity: 
the  broad  intent  is  to  include  government  engineering  centers, 
laboratories,  arsenals,  federally  funded  research  centers, 
educational  institutions,  not-for-profit  corporations,  and 
profit-oriented  firms  that  do  not  manufacture  or  produce  hard¬ 
ware  or  computer  software.  The  principal  distinction  is  that 
this  group  usually  lacks  the  insight  into  the  discrete  segment 
of  the  industrial  economy  that  will  ultimately  dictate  the 
production  price  of  the  end  item  and  create  or  inhibit  a  com¬ 
petitive  environment.  Further,  a  technology  transfer  must 
occur  whenever  the  transition  from  non-industrial  firms  to  in¬ 
dustrial  firms  occurs. 
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2.3  DESCRIPTION  OF  SCENARIO 

2.3.1  Category  Selection 

The  prototype  model  consists  of  three  segments,  dis¬ 
played  in  Figure  2.3-1.  The  objective  of  the  first  phase  is 
to  build  a  user-defined  baseline  scenario.  This  is  accomplish¬ 
ed  through  a  series  of  computer-generated  questions  and  user 
responses  selected  from  a  menu.  The  first  question  concerns 
the  category  of  the  weapon  system  concept.  For  the  prototype 
procedure,  there  are  three  options: 

•  Development  of  a  new  tactical  missile 
system 

•  Development  of  a  major  modification  to 
an  existing  tactical  missile 

•  Development  of  a  key  electronic  subsystem. 

2.3.2  Setting  the  Stage 

The  next  question  asks  when  the  analysis  is  to  begin. 
For  the  prototype  model,  it  is  recommended  that  the  analysis 
begin  with  either  Phase  1  or  Phase  2.  Beginning  the  analysis 
with  Phase  0  is  permitted  but  not  recommended  for  reasons  dis¬ 
cussed  in  2.2.1.  If  the  user  is  interested  only  in  production 
alternatives,  indepth  analysis  with  program  specific  parameters 
is  far  superior  to  the  general  relationships  included  in  the 
model.  Accordingly,  beginning  the  analysis  with  Phase  3  is 
not  an  option. 

If  the  user  response  begins  the  analysis  with  Phase  1 
or  Phase  2,  the  computer  responds  with  questions  concerning 
the  strategy  alternatives  pursued  in  the  prior  phases,  togeth¬ 
er  with  the  time  and  cost  of  each  alternative.  There  are  two 
principal  reasons  for  these  questions: 
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•  All  combinations  of  the  phase  alter¬ 
natives  are  not  realistically  feasible. 
Knowledge  of  the  strategy  pursued  in 
prior  phases  enables  the  model  to  limit 
its  analysis  to  only  those  combinations 
which  are  feasible 

•  Requiring  the  user  to  identify  the  alter¬ 
natives  pursued  in  prior  phases  together 
with  the  time  and  cost  involved  rein¬ 
forces  the  concept  that  acquisition  stra¬ 
tegy  does  encompass  the  entire  acqui¬ 
sition  process;  thus  emphasizing  that 
the  selection  of  an  acquisition  alter¬ 
native  for  one  phase  should  not  be  made 
independently  of  other  phase  options. 


Next,  the  user  can  eliminate  from  consideration  alter¬ 
natives  which  may  not  be  feasible  (or  acceptable)  for  his  speci¬ 
fic  situation,  although  feasible  for  weapon  systems  development 
in  general.  Alternatives  should  be  eliminated  at  this  point 
only  if  it  would  be  impossible  to  (or  he  would  not  be  permit¬ 
ted  to)  execute  that  alternative  should  it  appear  attractive. 


2.3.3  Perception  of  Technical  Risks 

The  next  series  of  questions  are  crucial  to  the  des¬ 
cription  of  the  baseline  scenario.  These  questions  are  de¬ 
signed  to  capture  the  user's  perception  of  the  technical  risk 
associated  with  developing  his  weapon  system  concept.  In  this 
model,  technical  risk  is  separated  into  three  distinct  cate¬ 
gories  : 


•  Level  of  technology  advance  -  the  con 
cept  embodied  in  this  category  is  the 
magnitude  of  the  technology  increase 
over  the  existing  state  of  the  art 
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•  Degree  of  required  system  integration  - 
a  large  weapon  system  with  many  complex 
internal  and  external  interfaces  is  a 
high-technology  risk  program;  not  neces¬ 
sarily  because  it  embodies  advanced  tech¬ 
nology,  but  because  it  is  vulnerable  to 

a  large  number  of  error  sources 

•  Level  of  software  dependency  and  com¬ 
plexity  -  a  weapon  system  using  off- 
the-shelf  components  with  few  interfaces 
may  still  be  dependent  on  a  large  and 
complex  computer  software  development 
effort.  If  the  software  is  critical  to 
the  operation  of  the  system,  its  develop¬ 
ment  could  pace  the  development  of  the 
entire  system. 


For  each  of  the  three  technical  risk  categories,  the 
user  is  asked  for  the  best  estimate  of  his  program's  level  of 
risk  at  the  start  of  the  analysis  (e.g.,  if  he  stated  pre¬ 
viously  that  the  analysis  was  to  begin  with  Phase  1,  the  risk 
estimates  should  correspond  to  that  time).  For  each  category, 
the  levels  of  risk  are  expressed  on  an  arbitrary  scale  from 
one  to  nine.  A  one  corresponds  to  virtually  no  risk,  while  a 
nine  corresponds  to  maximum  risk.  Definitions  have  been  pro¬ 
vided  for  intermediate  points  along  the  scale  (Table  2.3-1). 
However,  these  definitions  are  not  cast  in  concrete.  The 
intent  is  to  provide  a  guide  to  aid  the  user.  If  the  user 
feels  uncomfortable  with  these  definitions,  his  own  intuition 
should  be  used.  The  relative  placement  on  the  scale  of  his 
program  risk  given  the  scale  provided  is  the  important  aspect. 
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Intermediate  Values 
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In  addition  to  estimating  the  risk  in  each  category, 
the  user  is  asked  for  his  degree  of  confidence  in  that  esti¬ 
mate.  Again,  this  is  defined  by  an  arbitrary  scale  from  one 
to  nine  where  one  stands  for  total  uncertainty  and  nine  stands 
for  total  certainty.  Definitions  are  provided  for  inter¬ 
mediate  points  (Table  2.3-2).  The  purpose  of  including  this 
aspect  is  to  include  uncertainty  in  the  model. 


TABLE  2.3-2 
DEGREES  OF  CONFIDENCE 


Level 

Definition 

9 

Absolutely  certain 

7 

Reasonably  certain 

5 

Sounds  reasonable 

3 

Somewhat  unsure 

1 

Total  uncertainty 

■C' 

O' 

00 

intermediate  Values 

2. 3. A  Estimating  Inventory  Requirements 

The  next  input  to  the  baseline  scenario  is  an  esti¬ 
mate  of  inventory  requirements.  This  is  expressed  as  the  num¬ 
ber  of  systems  to  be  produced  and  the  number  of  years  of  pro¬ 
duction  (e.g.,  5000  systems  to  be  produced  over  7  years).  The 
user  should  provide  this  information  for  a  low  quantity  esti¬ 
mate,  a  moderate  quantity  estimate,  and  a  high  quantity  esti¬ 
mate.  This  permits  the  model  to  include  uncertainty  in  pro¬ 
jected  inventory  requirements  defined  during  early  development 
periods.  The  relative  ability  of  the  strategies  to  accomodate 
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different  quantity  estimates  is  captured  in  the  model.  If 
there  is  a  significant  variation  among  the  inventory  estimates, 
this  influences  the  desirability  of  the  strategies,  and  it 
will  be  reflected  in  the  results  obtained. 

2.3.5  Estimating  Relative  Production  Cost 

The  final  input  to  the  baseline  scenario  is  an  esti¬ 
mate  of  the  system's  relative  production  cost  compared  to  all 
systems  in  that  weapon  system  category*  (for  the  prototype 
model,  the  two  categories  are  tactical  missiles  and  key  elec¬ 
tronic  subsystems).  Again,  an  arbitrary  scale  from  one  to 
nine  is  used.  Definitions  are  provided  in  Table  2.3-3.  As 
before,  the  user  is  asked  for  his  degree  of  confidence  in  this 
estimate  using  the  previously  defined  scale  (Table  2.2-2). 
This  concludes  the  first  segment  of  the  model. 


TABLE  2.3-3 

SCALE  FOR  RELATIVE  PRODUCTION  COST  ESTIMATE 


Level 

Definition 

9 

Well  above  average 

7 

Above  average 

5 

Average 

3 

Below  average 

1 

Well  below  average 

2,  4,  6,  8 

Intermediate  Values 

*  This  subjective  assessment  of  production  cost  was  chosen 
over  a  more  precise  indicator  (such  as  first  unit  cost  or 
average  cost  over  the  production  run)  primarily  for  simpli¬ 
city.  Early  in  the  development  of  a  weapon  system,  a  user 
of  the  model  is  more  likely  to  have  a  feeling  that  his  pro¬ 
duction  costs  are  going  to  be  relatively  expensive  (or  inex¬ 
pensive)  compared  to  similar  weapon  systems  than  he  is  to 
know  a  precise  value  in  some  base  year  dollars. 
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2.4  GENERATION  OF  STRATEGIES  AND  CALCULATION  OF  ATTRIBUTES 

This  segment  of  the  model  requires  no  user  involve¬ 
ment.  Typically,  the  user  has  to  wait  for  a  few  seconds  while 
the  computer  performs  several  calculations .  Based  upon  the 
input  provided  during  the  prior  phase,  the  model  generates  a 
set  of  allowable  acquisition  strategies  and  computes  several 
attributes  for  each,  principally: 

•  A  probability  distribution  of  the  time 
required  to  reach  initial  operational 
capability  (IOC) 

•  A  probability  distribution  of  the  devel¬ 
opment  cost 

•  A  probability  distribution  of  production 
cost  for  each  input  estimate  of  inventory 
requirements 

•  Probability  distributions  of  the  tech¬ 
nical  risk  remaining  in  each  category  at 
the  completion  of  FSD. 

Once  this  is  completed,  the  model  outputs  the  number  of  strat¬ 
egies  generated  and  gives  the  user  the  option  to  save  the  in¬ 
formation  for  future  reference. 


2.5  EVALUATION  OF  ALTERNATIVES 
2.5.1  Probability  of  Success 

The  first  step  in  the  evaluation  segment  is  to  reduce 
the  set  of  strategies  based  on  the  probability  of  meeting  the 
perceived  urgency  of  need.  To  accomplish  this,  the  user  in¬ 
puts  two  estimates:  the  number  of  months  from  the  beginning 
of  his  initial  phase  to  the  earliest  desired  IOC  and  the  number 
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of  months  to  the  latest  acceptable  IOC.  This  approach  was 
chosen  to  bound  the  urgency  of  need  concept.*  The  model  then 
displays  the  relative  capability  of  the  strategies  to  success¬ 
fully  meet  these  specified  IOC  requirements  (see  Figure  2.5-1) 
The  three  levels  identified  on  the  left  of  each  table  corre¬ 
spond  to  three  arbitrarily  defined  pre-production  design  sta¬ 
bility  indicators.  The  following  definitions  are  provided  to 
indicate  the  intent  of  the  categorization: 

•  Level  1  -  Virtually  all  technical  risks 
have  been  eliminated.  The  system  is  ready 
for  mass  production 

•  Level  2  -  Minor  technical  problems  per¬ 
sist,  but  minor  system  modifications 
during  production  should  resolve  them 

•  Level  3  -  Somewhat  more  significant  tech¬ 
nical  problems  remain.  Limited  produc¬ 
tion  only  (perhaps  in  conjunction  with 
planned  product  improvement)  is  recom¬ 
mended  . 

The  numbers  along  the  top  of  each  column  (0.10,  0.20, 
etc.)  are  probabilities  of  success.  They  indicate  the  proba¬ 
bility  of  successfully  achieving  the  IOC  requirement  with  a 
specified  level  of  design  stability.  An  entry  in  the  table 
corresponds  to  the  number  of  strategies  which  have  a  proba¬ 
bility  at  least  as  great  as  that  indicated  of  successfully 
meeting  the  IOC  requirement  at  the  indicated  level  of  design 
stability.  For  example,  in  Table  I  of  Figure  2.5-1,  the 
number  48  appears  in  the  third  row  (representing  Level  3) 


*Another  consideration  influencing  this  approach  stemmed  from 
the  historical  review.  It  was  common  for  the  IOC  requirements 
stated  early  in  the  development  effort  to  be  substantially 
modified.  Actual  times  required  to  reach  IOC  were  typically 
much  longer  than  those  desired. 
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Table  I 

-  Earliest  Desired  IOC 

Probability  of 

Success 

at  Least  as 

Great  as: 

0.10 

0.20 

0.30 

0.40 

0.50 

0.60 

0.70  0.80 

0.90 

Level 

1 

186 

168 

108 

90 

30 

18 

0  0 

0 

Level 

2 

228 

186 

168 

108 

90 

30 

18  0 

0 

Level 

3 

246 

228 

186 

168 

108 

90 

48  0 

0 

Table  II 

-  Latest  Acceptable 

IOC 

Probability  of 

Success 

at  Least  as 

Great  as: 

0.10 

0.20 

0.30 

0.40 

0.50 

0.60 

0.70  0.80 

0.90 

Level 

1 

186 

168 

108 

90 

30 

30 

0  0 

0 

Level 

2 

228 

186 

168 

108 

90 

30 

30  0 

0 

Level 

3 

246 

228 

186 

168 

108 

90 

60  30 

0 

Figure  2.5-1  Probability  of  Success 


under  the  column  headed  by  0.70.  Thus,  there  are  48  strat¬ 
egies  that  can  successfully  meet  the  earliest  desired  IOC  with 
a  probability  of  at  least  0.70  at  design  stability  Level  3  or 
better.  Note  that  if  one  were  unwilling  to  accept  Level  3 
design  stability  and  require  a  more  stable  design,  there  are 
less  strategies  available.  Only  18  strategies  have  a  proba¬ 
bility  of  success  of  at  least  0.70  at  Level  2  and  there  are 
no  strategies  with  a  probability  of  success  as  high  as  0.70 
at  design  stability  Level  1. 


From  this,  the  user  is  asked  to  eliminate  those  stra¬ 
tegies  whose  probabilities  of  success  are  unsatisfactorily  low 
(or  conversely,  to  select  for  further  consideration  only  those 
strategies  whose  probabilities  of  success  are  satisfactorily 
high).  This  is  accomplished  by  specifying  a  minimum  accept¬ 
able  probability  of  success  for  each  IOC  and  level  of  design 
stability.  For  the  example  displayed  in  Figure  2.2-3,  one 
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might  choose  to  enter  0.5,  0.6,  and  0.7  as  the  minimum  accept¬ 
able  probabilities  for  Levels  1,  2,  and  3,  respectively,  for 
the  earliest  desired  IOC,  and  0.6,  0.7,  0.8  as  the  minimum 
acceptable  probabilities  for  the  latest  acceptable  IOC  for 
levels  1,  2,  and  3,  respectively. 

This  input  corresponds  to  six  criteria  used  for  stra¬ 
tegy  comparison.  It  is  common  for  the  strategies  which  maxi¬ 
mize  the  probability  of  success  for  the  early  IOC  to  be  dis¬ 
tinct  from  the  strategies  which  maximize  the  probability  of 
success  for  the  late  IOC.  Because  of  this  phenomenon,  the 
model  responds  with  a  summary  of  the  results  obtained  by  evalu 
ating  the  strategies  against  the  six  criteria.  The  format  for 
the  summary  is  as  follows: 

Nj  strategies  satisfy  all  six  criteria 

N2  additional  strategies  satisfy  5  criteria  and  are 
'  within  a  probability  of  0.10  on  the  sixth 

No  additional  strategies  satisfy  4  criteria  and  are 
J  within  a  probability  of  0.10  on  the  other  two 

additional  strategies  satisfy  3  criteria  and  are 
within  a  probability  of  0.10  on  the  other  3 

The  notation  ,  N2 ,  N^,  and  stand  for  the  number  of  strat¬ 
egies  in  each  category.  Depending  on  how  the  minimum  accept¬ 
able  probabilities  were  specified,  it  is  common  for  some  of 
the  Nj  through  to  be  zero.  (There  even  are  occasions  when 
all  four  are  zero).  Accordingly,  the  user  is  given  the  option 
to  specify  other  minimum  acceptable  probabilities  and  reeval¬ 
uate  the  strategies  against  a  new  set  of  criteria.  This  pro¬ 
cess  may  be  iterated  until  the  user  is  satisfied  with  the  re¬ 
sults  obtained. 
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Once  reasonable  satisfaction  with  these  results  is 
obtained,  the  user  has  four  options  available  for  selecting 
the  strategies  to  consider  for  further  analysis: 


•  Consider  only  those  N,  strategies  that 
satisfy  all  six  criteria 

•  Consider  those  N,  +  N~  strategies  that 
satisfy  at  least1 f ive^criteria  and  are 
"close"  (i.e.,  within  a  probability  of 
0.10)  on  the  sixth 

•  Consider  those  N,  +  N2  +  N^  strategies 
that  satisfy  at  least  four^criteria  and 
are  "close"  on  the  other  two 

•  Consider  those  N,  +  +  +  N,  strat¬ 

egies  that  satisiy  at^least  three  crite¬ 
ria  and  are  "close"  on  the  other  three. 


Once  this  option  is  specified,  the  model  then  prints 
a  summary  of  the  phase  alternatives  in  the  strategies  selected 
for  further  analysis.  An  example  of  the  format  used  in  this 
summary  is  displayed  in  Figure  2.5-2. 


The  following  summarizes  the  number  of 

the  30  stra- 

tegies  remaining 

for  analysis  which  include  the 

indicated 

alternatives : 

FOR  PHASE  1: 

Prototype  Non- industrial 

12 

Prototype  Single  Industrial 

6 

Prototype  Multiple  Industrial 

12 

FOR  PHASE  2: 

Incremental  -  Single  Source 

18 

Incremental  -  Multiple  Sources 

12 

FOR  PHASE  3: 

Single  Source  -  No  Options 

5 

Single  Source  -  Options 

5 

Single  Source  -  MYC 

5 

Licensing 

5 

Leader/Follower 

5 

Second  Sourcing 

5 

Figure  2.5-2  Summary  of  Phase  Alternatives 
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2.5.2  Relative  Cost  Considerations 

The  next  step  in  the  evaluation  process  considers  the 
differences  in  expected  cost  among  the  remaining  strategies. 
The  computer  first  displays  three  tables  such  as  those  pre¬ 
sented  in  Figure  2.5-3.  The  tables  compare  development  cost 
to  total  program  cost,  in  a  relative  sense,  for  each  of  the 
low,  moderate,  and  high  estimated  inventory  requirements.  The 
horizontal  scale  in  each  table  is  normalized  to  the  highest 
development  cost  (expected  value  plus  one  standard  deviation) 
associated  with  the  strategies  under  consideration.  The  ver¬ 
tical  scale  in  each  table  is  normalized  to  the  highest  total 
program  cost  (expected  value  plus  one  standard  deviation)  asso¬ 
ciated  with  the  strategies  under  consideration  for  that  inven¬ 
tory  requirement.  An  entry  in  the  table  represents  the  number 
of  strategies  whose  expected  development  cost  and  total  pro¬ 
gram  cost  lie  in  the  range  indicated.  For  example,  in  the 
table  representing  the  moderate  production  quantity  in  Figure 
2.5-3,  there  are  eight  strategies  (out  of  the  total  of  30 
remaining)  whose  expected  development  cost  is  between  30%  and 
40%  of  the  maximum  and  whose  total  program  cost  is  between  90% 
and  100%  of  the  maximum. 

Tradeoffs  frequently  exist  among  development  cost  and 
production  cost.  If  there  is  considerable  variation  among  the 
low,  moderate,  and  high  inventory  estimates,  some  of  these 
tradeoffs  may  be  readily  apparent.  Others  may  be  more  subtle. 

The  user  is  now  given  the  opportunity  to  eliminate 
additional  strategies  from  consideration  based  upon  these  re¬ 
lative  cost  comparisons.  This  is  accomplished  in  a  manner 
similar  to  that  employed  with  specifying  minimum  acceptable 
probabilities . 
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In  this  case,  the  input  is  maximum  acceptable  rela¬ 
tive  cost  for  development  and  for  total  program  cost  corre¬ 
sponding  to  the  three  inventory  estimates.  For  example,  in 
Figure  2.5-3,  a  user  may  specify  0.5  as  the  maximum  relative 
development  cost  if  his  R&D  funds  are  severly  limited.  If  a 
limitation  on  R&D  funds  is  not  a  driving  concern,  he  may  choose 
to  enter  1.0  which  eliminates  development  cost  as  a  signi¬ 
ficant  attribute.  Maximum  total  program  cost  is  specified  in 
a  similar  manner.  For  example,  0.7.,  0.9,  and  0.8  may  be  rea¬ 
sonable  input  for  the  example  in  Figure  2.5-3  for  the  low, 
moderate,  and  high  inventory  estimates,  respectively. 

This  input  corresponds  to  four  criteria  upon  which 
the  strategies  are  compared.  Results  are  displayed  in  a 
manner  similar  to  that  used  for  probabilities  of  success: 

Nj  strategies  satisfy  all  four  criteria 

additional  strategies  satisfy  3  criteria  and  are 
within  10%  on  the  fourth 

No  additional  strategies  satisfy  2  criteria  and  are 
within  10%  on  the  other  two 

The  notation  N^ ,  N2 .  and  stand  for  the  number  of  strategies 
in  each  category.  Agal..,  it  is  common  for  some  of  the  N^ 
through  N^  to  be  zero.  Accordingly,  the  user  is  given  the 
option  to  select  other  maximum  acceptable  relative  costs  and 
reevaluate  the  strategies  with  the  new  set  of  criteria.  This 
process  may  be  repeated  until  the  user  is  satisfied  with  the 
results . 


Once  reasonable  satisfaction  is  obtained,  the  user 
has  three  options  for  selecting  the  strategies  for  further 
analysis : 
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•  Consider  only  those  N,  strategies  that 
satisfy  all  four  criteria 

•  Consider  those  N,  +  N~  strategies  that 
satisfy  three  criteria  and  are  "close" 
on  the  fourth 

•  Consider  those  N,  +  +  No  strategies 

which  satisfy  two  criteria-3 and  are  "close" 
on  the  other  two. 

Once  this  option  is  specified,  the  model  displays  the 
number  of  strategies  eliminated  (if  any)  and  gives  the  user 
the  option  of  displaying  those  strategies  eliminated  as  well 
as  listing  those  strategies  remaining. 

2.5.3  Dominance  Analysis 

This  portion  of  the  evaluation  requires  no  user  in¬ 
volvement.  The  remaining  strategies  are  compared  to  each  other 
to  determine  any  "second-best"  choices.  Basically,  one  strat¬ 
egy  is  considered  to  dominate  another  if  it  is  clearly  superior 
to  the  other  strategy  in  at  least  one  (or  more)  attribute(s) 
and  at  least  as  good  on  all  others.  This  comparison  is  not 
performed  in  an  absolute  sense.  Rather,  two  attributes  are 
considered  to  be  equivalent  if  they  are  reasonably  close  to 
each  other.  Similarly,  for  one  attribute  to  be  considered  su¬ 
perior  to  another,  it  must  be  better  by  more  than  a  prespec¬ 
ified  threshhold.  Combining  these  two  considerations  into  a 
single  algorithm  is  somewhat  unique  and  was  implemented  in  an 
experimental  fashion.  Further  research  into  the  full  implica¬ 
tion  of  the  approach  is  justified.  More  details  of  the  method¬ 
ology  are  provided  in  Appendix  A. 
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Once  the  dominance  analysis  is  completed,  the  computer 
displays  the  number  of  strategies  eliminated.  The  user  then 
has  the  option  of  displaying  those  strategies. 


2 . 6  RESULTS 

The  results  of  the  analysis  consist  of  strategies  not 
eliminated  by  one  or  more  criteria.  Typically,  only  a  small 
number  of  strategies  remain.  These  are  displayed,  together 
with  the  following  principal  attributes: 

•  Probability  of  success  by  the  earliest 
desired  IOC  by  level  of  design  stability 

•  Probability  of  success  by  the  latest  ac¬ 
ceptable  IOC  by  level  of  design  stability 

•  Relative  development  cost 

•  Relative  total  program  cost  by  low, 
moderate,  and  high  inventory  estimates. 

A  sample  of  the  display  is  provided  in  Figure  2.6-1. 
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Lgure  2.6-1  Summary  of  Results 
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3. 


EXAMPLES 


This  chapter  discusses  two  examples.  The  first  is 
hypothetical  and  it  includes  variations  indicative  of  sensi¬ 
tivity  analyses  potentially  useful  in  a  teaching  environment 
The  second  example  is  an  application  of  the  model  to  a  real 
program  with  all  inputs  provided  by  the  program  manager. 


3.1  AN  ILLUSTRATIVE  EXAMPLE 


The  example  described  in  this  section  is  purely  hy¬ 
pothetical.  The  intent  of  this  section  is  to  demonstrate  the 
operation  of  the  model  and  to  illustrate  how  user-supplied  in¬ 
puts  during  the  evaluation  segment  affect  the  results. 

3.1.1  Baseline  Scenario 


This  example  concerns  the  development  of  a  new  tacti¬ 
cal  missile  system.  The  weapon  system  concept  was  directed 
(concept  exploration  phase  was  skipped),  and  the  analysis  is 
to  begin  with  Phase  1  -  Demonstration  and  Validation.  All 
strategy  alternatives  included  in  the  model  are  considered 
feasible.  Parameters  defining  the  perception  of  technical  risk 
are  as  follows: 


Risk  Category  Risk 

Technology  Advance  7  - 

System  Integration  7  - 

Software  Dependency  7  - 

and  Complexity 


Level  Confidence  Level 


Major 

7 

-  Reasonably 

Certain 

Major 

7 

-  Reasonably 

Certain 

Ma  jor 

9 

-  Absolutely 

Certain 
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Estimates  of  inventory  requirements  were  specified  with  a  wide 
variation  so  that  tradeoffs  dependent  upon  the  actual  quantity 
should  exist.  Parameters  provided  are  as  follows: 

•  Low  estimate  -  2000  systems  produced 
over  2  years 

•  Moderate  estimate  -  20,000  systems  pro¬ 
duced  over  8  years 

•  High  estimate  -  60,000  systems  produced 
over  15  years. 

Relative  production  costs  were  estimated  as  average  (5)  with  a 
confidence  level  of  5  (sounds  reasonable). 

From  this  input,  264  possible  acquisition  strategies 
were  generated. 

3.1.2  Baseline  Evaluation 


Earliest  desired  IOC  and  latest  acceptable  IOC  esti¬ 
mates  of  144  months  (12  years)  and  168  months  (14)  years  were 
provided.  While  this  time  frame  may  seem  excessive,  it  is 
not  uncommon  in  weapon  system  development  programs,  especially 
where  high  technology  programs  are  concerned.  Probabilities 
of  success  generated  are  displayed  in  Figure  3.1-1.  The  mini¬ 
mum  acceptable  probabilities  selected  are  circled. 
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TABLE  1 

EARLIEST  DESIRED  IOC 

PROBABILITY  OF  SUCCESS  AT  LEAST  AS  GREAT  AS: 


0.10 

0.20 

0.30 

0.40 

0.50 

0.60 

0.70 

0.80 

0.90 

LEVEL  1 

186 

168 

108 

90 

(30) 

Tg 

0 

0 

0 

LEVEL  2 

228 

186 

168 

108 

90 

( 30) 

IB 

0 

0 

LEVEL  3 

246 

228 

186 

168 

108 

90 

(48) 

0 

0 

TABLE  2 

LATEST  ACCEPTABLE  IOC 

PROBABILITY  OF  SUCCESS  AT  LEAST  AS  GREAT  AS: 


0.10 

0.20 

0.30 

0.40 

0.50 

0.60 

0.70 

0.80 

0.90 

LEVEL 

1 

186 

168 

108 

90 

30 

(30) 

0 

0 

0 

LEVEL 

2 

228 

186 

168 

108 

90 

50 

(30) 

0 

LEVEL 

3 

246 

228 

186 

168 

108 

90 

60 

60) 

0 

Figure  3.1-1  Probabilities  of  Success 


The  results  of  applying  these  criteria  were  that  30 
strategies  satisfied  all  six  criteria.  The  other  "close"  cat¬ 
egories  were  all  empty  (i.e.,  contained  zero  strategies).  The 
summary  of  these  30  strategies  by  phase  is  as  follows: 


FOR  PHASE  1: 

Prototype  Nonindustrial 

12 

Prototype  Single  Industrial 

6 

Prototype  Multiple  Industrial 

12 

FOR  PHASE  2: 

Incremental  -  Single  Source 

18 

Incremental  -  Multiple  Sources 

12 

FOR  PHASE  3:  Single  Source  -  No  Options 

Single  Source  -  Options 

Single  Source  -  MYC 

Licensing 
Leader/Follower 
Second  Sourcing 


5 

5 

5 

5 

5 

5 


3-3 


THE  ANALYTIC  SCIENCES  CORPORATION 


Relative  cost  comparisons  of  these  30  strategies  are 
provided  in  Tables  3.1-1  through  3.1-3.  It  was  decided  not  to 
eliminate  any  of  the  strategies  at  this  point  for  cost  con¬ 
siderations,  primarily  to  discover  how  many  strategies  would 
be  eliminated  by  the  dominance  analysis. 


The  result  was  that  21  of  the  30  strategies  were 
eliminated  as  being  "second-best"  options.  The  output  summary 
of  results  is  displayed  in  Figure  3.1-2. 


The  implication  of  these  results  are  as  follows: 


•  For  the  high-risk  situation  described, 
when  urgency  of  need  is  not  great,  one 
should  build  a  full  system  prototype 
during  D&V  and  perform  incremental  FSD. 
Tradeoffs  exist  between  non- industrial 
and  industrial  firms,  as  well  as  single 
and  dual  source  FSD 

•  Production  options  are  mainly  dependent 
upon  estimated  inventory  requirements. 
The  high  estimate  justifies  dual  source 
production,  the  low  estimate  does  not. 
For  the  moderate  estimate,  there  are 
other  tradeoffs  involved 

•  The  probability  of  a  successful  program 
is  relatively  high. 
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Th«rc  are  9  strategies  reaainlng.  The  following  suaaa rites 
the  relative  attributes  of  these  strategies.  Coaplicated  trade-offs 
exist  aaong  these  attributes.  Trade-off  analysis  aore  detailed 
and  aore  in-depth  than  is  possible  here  is  recaaaended. 


1  Prototype 

2  Prototype 

3  Prototype 

4  Prototype 

5  Prototype 

6  Prototype 

7  Prototype 

8  Prototype 

9  Prototype 


PHASE  1 

Mon- industrial 
Mon-industrial 
Mon- industrial 
Mon-industrial 
Mon-industrial 
Mon-industrial 
Single  Industrial 
Single  Industrial 
Single  Industrial 


Increaental 

lncreaental 

Increaental 

Increaental 

Increaental 

Increaental 

Increaental 

Increaental 

Increaental 


PHASE  2 

-  Single  Source 

-  Single  Source 

-  Single  Source 

-  Multiple  Sources 

-  Multiple  Sources 

-  Multiple  Sources 

-  Single  Source 

-  Singe  Source 

-  Single  Source 


PHASE  3 

Single  Source  -  MYC 
Leader/To 1 lower 
Second  Sourcing 
Single  Source  -  HYC 
Leader/Fol lower 
Second  Sourcing 
Single  Source  -  MYC 
Leader/Follower 
Second  Sourcing 


s 

PROBABILITY  OF  SUCCESS 

RELATIVE  COSTS 

T 

* 

EARLY  IOC 

LATE  IOC 

DEV 

TOTAL 

• 

LI 

L2 

L3 

LI 

L2 

L3 

LQ 

HQ 

HQ 

l 

0.60 

0.68 

0.76 

0.64 

0.73 

0.82 

0.38 

0.66 

0.91 

0.94 

2 

0.60 

0.68 

0.76 

0.64 

0.73 

0.82 

0.36 

0.81 

0.84 

0.78 

3 

Q.60 

0.68 

0.76 

0.64 

0.73 

0.82 

0.38 

0.81 

0.84 

0.78 

4 

0.60 

0.68 

0.76 

0.64 

0.73 

0.82 

0.68 

0.63 

0.73 

0.81 

S 

0.60 

0.68 

0.76 

0.64 

0.73 

0.82 

0.68 

0.84 

0.76 

0.69 

6 

0.60 

0.68 

0.76 

0.64 

0.73 

0.82 

0.68 

0.84 

0.76 

0.69 

1 

0.62 

0.71 

0.79 

0.64 

0.74 

0.82 

0.39 

0.66 

0.91 

0.94 

8 

0.62 

0.71 

0.79 

0.64 

0.74 

0.82 

0.39 

0.81 

0.84 

0.78 

9 

0.62 

0.71 

0.79 

0.64 

0.74 

0.82 

0.39 

0.81 

0.84 

0.78 

Please 

!  note 

—  The 

use  of  aultiyear  contractinj 

1  assumes 

a  relatively  stable  design  for  initial  production.  If  this 
Is  not  the  case,  single  source  with  or  without  options  would 
be  the  alternatives. 


Figure  3.1-2  Summary  of  Results 
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3.1.3  Variation  A  --  Increased  Urgency  of  Need 

The  same  baseline  was  reevaluated  under  an  extreme 
urgency-of-need  situation  for  such  a  high  technology  program. 
Earliest  desired  IOC  and  latest  acceptable  IOC  inputs  were  72 
months  (6  years)  and  96  months  (8  years),  respectively.  The 
resultant  probability  of  success  tables  are  displayed  in  Figure 
3.1-3.  Minimum  acceptable  probability  specifications  are 
circled.  Not  surprisingly,  the  probabilities  of  success  are 
greatly  reduced. 


TABLE  1 

EARLIEST  DESIRED  IOC 


PROBABILITY  OF  SUCCESS  AT  LEAST  AS  GREAT  AS: 


0. 10 
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LEVEL 
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3 
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0 
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0 
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0 

TABLE 

2 

LATEST 

ACCEPTABLE 

IOC 

PROBABILITY  OF  SUCCESS  AT  LEAST  AS  GREAT  AS: 
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LEVEL 

1 
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152 
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The  result  of  applying  these  criteria  was  as  follows: 


•  18  Strategies  satisfy  all  6  criteria 

•  18  Additional  strategies  satisfy  5 
criteria  and  are  within  a  probability  of 
.10  on  the  sixth 

•  18  Additional  strategies  satisfy  4 
criteria  and  are  within  a  probability  of 
.10  on  the  other  two 

•  12  Additional  stategies  satisfy  3 
criteria  and  are  within  a  probability 
within  of  .10  on  the  other  three 


The  top  two  categories  (consisting  of  36  strategies) 
were  selected  for  further  analysis.  The  summary  by  phase  of 
these  36  strategies  is  as  follows: 


FOR  PHASE  1:  Prototype  Non- industrial  6 

Prototype  Single  Industrial  12 

Prototype  Multiple  Industrial  18 

FOR  PHASE  2:  Full  Concurrency  -  Single  Source  18 

Partial  Concurrency  -  Single  Source  12 

Partial  Concurrency  -  Multiple  Sources  6 


FOR  PHASE  3:  Single  Source  -  No  Options 

Single  Source  -  Options 

Single  Source  -  MYC 

Licensing 
Leader/Follower 
Second  Sourcing 


Relative  cost  comparisons  are  presented  in  Table  3.1-4 
through  3.1-6.  Again,  no  strategies  were  eliminated  based 
upon  relative  cost.  Dominance  analysis  eliminated  27  of  the 
36  strategies.  A  summary  of  the  results  is  displayed  in 
Figure  3.1-4. 
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Implications  of  these  results  are  as  follows: 


•  Even  with  the  rather  extreme  urgency  of 
need  specified,  a  program  of  relatively 
high  technology  justifies  a  full  system 
prototype  during  D&V 

•  A  certain  amount  of  concurrency  is  man¬ 
dated  by  the  urgency  of  need 

•  Production  options  are  again  principally 
a  function  of  inventory  estimates 

•  Probability  of  success  is  greatly  reduced. 


3.1.4  Variation  B  --  Increased  Urgency  of  Need  and 
Alternate  Selection  Criteria 


The  same  situation  as  Variation  A  is  evaluated.  How¬ 
ever,  this  time  the  top  three  categories  resulting  from  apply¬ 
ing  the  probability  of  success  criteria  (consisting  of  54  stra¬ 
tegies)  are  selected.  The  summary  of  the  strategies  by  phase 
is  as  follows: 


FOR  PHASE  1:  Subsystem  Dev  Single  Industrial  6 

Subsystem  Dev  Multiple  Industrial  12 

Prototype  Non- industrial  6 

Prototype  Single  Industrial  12 

Prototype  Multiple  Industrial  18 

FOR  PHASE  2:  Partial  Concurrency  -  Single  Source  24 

Partial  Concurrency  -  Multiple  Sourrco  12 
Full  Concurrency  -  Single  Source  18 

FOR  PHASE  3:  Single  Source  -  No  Options  9 

Single  Source  Options  9 

Single  Source  -  MYC  9 

Licensing  9 

Leader/Follower  9 

Second  Sourcing  9 
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This  relaxation  in  success  criteria  resulted  in  subsystem  dev¬ 
elopment  becoming  an  option  during  D&V.  Again,  no  strategies 
were  eliminated  on  cost  consideration;  however,  dominance  ana¬ 
lysis  eliminated  39  of  the  54  strategies.  A  summary  of  the  15 
remaining  strategies  are  displayed  in  Figure  3.1-5. 

The  implications  are  that  subsystem  development  dur¬ 
ing  D&V  combined  with  partial  currency  during  FSD  become  a  vi¬ 
able  option  under  the  scenario  described.  The  probability  of 
success  is  lowered,  but  not  drastically. 

3.1.5  Variation  C  --  Limited  R&D  Funding 

This  variation  consists  of  the  baseline  scenario  with 
the  baseline  evaluation  (Section  3.1.2)  up  through  the  rela¬ 
tive  cost  comparisons  presented  in  Tables  3.1-1  through  3.1-3. 
This  evaluation  was  performed  under  the  assumption  of  limited 
R&D  funds  and  a  .5  was  entered  as  maximum  development  cost. 

No  constraints  were  placed  on  total  program  cost.  Of  the  18 
strategies  satisfying  this  criteria,  12  were  eliminated  via 
dominance  analysis.  A  summary  of  the  six  remaining  strategies 
are  displayed  in  Figure  3.1-6. 

This  one  added  criteria  thus  restricted  development 
options  to  single  source  development.  With  this  reduction,  it 
should  be  noted  that  under  the  moderate  and  high  inventory 
estimates,  total  program  cost  is  increased. 


3.2  A  REAL-WORLD  EXAMPLE 

The  AN/SLQ-32  Shipboard  Electronic  Warfare  System 
program  began  in  October  1971  with  the  objective  of  developing 
and  procuring  a  coherent  series  of  electronic  warfare  systems 
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Hun  are  IS  strategies  regaining.  Tke  following 
Murltti  the  ralatlva  attributes  of  tkaaa  atrataglaa.  Coapll- 
cated  trade-offs  aaiat  aw|  tkaaa  attributes.  Trada-off  aw- 
lyses  aora  detailed  end  aora  la-depth  tkaa  la  possible  kara  la 


recaniMnded . 

Phase  1 

1  Subeystaa  Dav  Slagle  Industrial 

2  Subeystaa  Dav  Slagle  laduatrlal 

3  Subeystaa  Dav  Slagle  laduatrlal 

4  Subeystaa  Dav  Multiple  Industrial 

5  Subeystaa  Dev  Multiple  Industrial 

6  Subeystaa  Dav  Multiple  laduatrlal 

7  Prototype  Single  Industrial 
g  Prototype  Single  Industrial 
I  Prototype  Single  Industrial 

10  Prototype  Single  Industrial 

11  Prototype  Single  Industrial 

12  Prototype  Single  Industrial 

13  Prototype  Multiple  Industrial 

14  Prototype  Multiple  Industrial 

15  Prototype  Multiple  Industrial 


Phase  2 

Partial  Concurrency  •  Single  Source 
Partial  Coacsrraacy  •  Slagle  Source 
Partial  Concurrency  •  Slagle  Source 
Partial  Concurrency  -  Multiple  Sources 
Partial  Concurrency  •  Multiple  Sourcea 
Partial  Concurrency  •  Multiple  Sources 
Partial  Concurrency  -  Single  Source 
Partial  Concurrency  -  Single  Source 
Partial  Concurrency  -  Single  Source 
full  Concurrency  -  Single  Source 
Pull  Concurrency  •  Single  Source 
Full  Concurrency  •  Single  Source 
Partial  Concurrency  -  Multiple  Sources 
Partial  Concurrency  -  Multiple  Sources 
Partial  Concurrency  •  Multiple  Sources 


Phase  3 

Single  Source  •  HYC 
Leads r/Fo 11 over 
Second  Sourcing 
Single  Source  •  HYC 
Leader/Follower 
Second  Sourcing 
Single  Source  -  HYC 
Leader/Follower 
Second  Sourcing 
Single  Source  -  MY C 
Leader/Fol lover 
Second  Sourcing 
Single  Source  ■  HYC 
Leeder/Fol lover 
Second  Sourcing 


s 

probability 

OF  SUCCESS 

RELATIVE  COSTS 

T 

ft 

early  ioc 

LATE  IOC 

DEV 

TOTAL 

• 

Ll 

L2 

L3 

Ll 

L2 

L3 

IQ 

MQ 

HQ 

i 

0.09 

0  13 

0.18 

0.20 

0.30 

0.41 

0.34 

0.6! 

0.90 

0.94 

2 

0.09 

0  13 

0  IS 

0.20 

0.30 

0.41 

0.34 

0.88 

0.86 

0.80 

3 

0  09 

0.13 

0.18 

0.20 

0.30 

0.41 

0.34 

0.88 

0.86 

0.80 

4 

0.09 

0.13 

0.18 

0.20 

0.30 

0.41 

0.61 

0.61 

0.71 

0.81 

5 

0.09 

0.13 

0  18 

0.20 

0.30 

0.41  ' 

0.61 

0.84 

0.75 

0.69 

S 

0.09 

0.13 

0.18 

0.20 

0.30 

0.41 

0.61 

0.84 

0.73 

0.69 

7 

0.13 

0  14 

0.19 

0.32 

0.60 

0.47 

0.42 

0.69 

0.91 

0.95 

8 

0.13 

0  16 

0.15 

0  32 

0.40 

0.47 

0.42 

0.91 

0.87 

0.80 

9 

0.13 

0  14 

0.19 

0.32 

0.40 

0.47 

0.42 

0.91 

0.87 

0.80 

10 

0.17 

0.23 

0.29 

0.29 

0.39 

0.51 

0.42 

0.69 

0.91 

0  95 

11 

0.17 

0.23 

0.29 

0.29 

0.39 

0.51 

0.42 

0.91 

0.90 

0.82 

12 

0.17 

0.23 

0.29 

0.29 

0.39 

0.51 

0.42 

0.91 

0.90 

0.82 

13 

0  13 

0.16 

0.19 

0.32 

0.40 

0.47 

0.78 

0.67 

0.73 

0.82 

14 

0  13 

0.16 

0.19 

0.32 

0.40 

0.47 

0.78 

0.91 

0.77 

0.70 

IS 

0.13 

0.16 

0.19 

0.32 

0.40 

0.47 

0.78 

0.91 

0.77 

0.70 

Pleaee  note  —  Ike  use  of  aulti-year  contracting  attunes  s 
relatively  stable  design  for  Initial  production.  If  this 
la  not  the  case,  single  aource  with  or  without  options 
would  be  tbe  alternatives. 


Figure  3.1-5  Summary  of  Results 
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There  are  6  strategies  remaining.  The  following  summarizes  the 
relative  attributes  of  these  strategies.  Complicated  trade-offs  exist 
among  these  attributes.  Trade-off  analysis  more  detailed  and  more 
in-depth  than  is  possible  were  is  recommended. 
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for  near  fleet-wide  installation.  The  program  manager  from 
program  inception  to  the  third  year  of  production  was  Capt. 
Robert  A.  Hullander,  U.S.N.,  Ret.  Capt.  Hullander  was  a  major 
contributor  to  the  prior  acquisition  strategy  model  feasi¬ 
bility  study.  During  the  development  of  the  prototype  model, 
he  participated  as  a  consultant  on  a  few  issues,  but  he  was 
not  aware  of  the  status  of  the  model  until  its  completion.  At 
that  time,  he  agreed  to  help  evaluate  the  prototype  model. 


3.2.1  Scenario 


After  a  test  run 
proceeded  to  describe  the 
parameters.  The  scenario 


through  the  model,  Capt.  Hullander 
AN/SLQ-32  scenario  in  terms  of  model 
described  is  as  follows: 


•  System  category  --  electronic  subsystem 

•  Concept  was  directed  --  no  concept  ex¬ 
ploration  was  performed 

•  Strategy  analysis  began  with  Phase  1 

•  Use  of  non- industrial  firms  was  not  feasible 
during  the  D&V  Phase 

•  All  alternatives  during  FSD  and  produc¬ 
tion  were  input  as  possible,  although 
there  were  political  considerations 
which  made  some  difficult 

•  Technical  risk 

Technology  Advance  -  moderate  (5) 
with  reasonable  certainty  (7) 

System  Integration  -  total  (9)  with 
absolute  certainty  (9) 

Software  Dependency  &  Complexity  -  mod¬ 
erate  (5)  with  absolute  certainty  (9) 
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•  Inventory  requirements  were  for  300  sys¬ 
tems  to  be  produced  over  4  years  with  no 
uncertainty  (i.e.,  low,  moderate,  and 
high  estimates  were  identical) 

•  Relative  production  costs  would  be  7 
(Above  average)  with  a  confidence  level 
of  7  (Reasonably  certain). 


Given  this  scenario,  174  possible  strategies  were  generated. 
3.2.2  Evaluation 


Urgency  of  need  was  input  as  60  months  (5  years)  as 
the  time  to  the  earliest  desired  IOC  and  78  months  (6^  years) 
as  the  time  to  the  latest  acceptable  IOC.  These  were  based 
upon  the  actual  5-year  time  span  the  program  office  used  as  a 
goal  and  the  achieved  time  to  IOC  of  slightly  over  6  years. 
Summaries  of  the  probability  of  success  are  displayed  in  Fig¬ 
ure  3.2-1.  Minimum  acceptable  probabilities  specified  are 
circled . 


At  this  point  Capt.  Hullander  expressed  some  surprise 
at  the  magnitude  of  the  probabilities  (i.e.  they  seemed  too 
low).  As  discussed  earlier,  this  is  not  surprising  due  to  the 
assumption  of  independence  in  their  calculation.  A  proper  ac¬ 
counting  for  dependence  among  stochastic  variables  is  always  a 
non-trivial  task.  This  situation  is  no  different.  Additional 
research  is  warranted  if  realistic  probabilities  are  desired. 
Howe'er,  as  a  relative  measure,  the  existing  probabilities  are 
reasonable . 
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TABLE  1 

EARLIEST  DESIRED  IOC 

PROBABILITY  OF  SUCCESS  AT  LEAST  AS  GREAT  AS: 
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TABLE  2 

LATEST  ACCEPTABLE  IOC 

PROBABILITY  OF  SUCCESS  AT  LEAST  AS  GREAT  AS: 
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Figure  3.2-1  Probabilities  of  Success 


The  results  of  evaluating  the  174  strategies  against 
the  input  criteria  are  as  follows: 


•  0  strategies  satisfy  all  6  criteria 

•  18  additional  strategies  satisfy  5  cri¬ 
teria  and  are  within  a  probability  of 
0.10  on  the  sixth 

•  12  additional  strategies  satisfy  4  cri¬ 
teria  and  are  within  a  probability  of 
0.10  on  the  other  two 

•  0  additional  strategies  satisfy  3  cri¬ 
teria  and  are  within  a  probability  of 
0.10  on  the  other  three. 
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All  30  strategies  accounted  for  were  selected.  The  summary  of 
these  strategies  by  phase  follows: 


FOR  PHASE  1:  CD  Single  Industrial  6 

CD  Multiple  Industrial  12 

Prototype  Single  Industrial  6 

Prototype  Multiple  Industrial  6 

FOR  PHASE  2:  Incremental  -  Single  Source  12 

Incremental  -  Multiple  Sources  6 

Full  Concurrency  -  Single  Source  12 

FOR  PHASE  3:  Single  Source  -  No  Options  5 

Single  Source  -  Options  5 

Single  Source  -  MYC  5 

Licensing  5 

Leader/Follower  5 

Second  Sourcing  5 


Since  the  inventory  estimates  were  identical ,  one 
table  is  sufficient  to  display  relative  cost  comparisons  (Table 
3.2-1). 


TABLE  3.2-1 

RELATIVE  COST  COMPARISON 
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Capt.  Hul lander  stated  he  would  prefer  not  to  eliminate  arbitrar¬ 
ily  any  alternatives  based  upon  cost.  Thus,  all  30  strategies 
were  subjected  to  dominance  analysis.  This  analysis  resulted 
in  26  of  the  30  strategies  being  eliminated.  The  26  "second- 
best"  strategies  are  displayed  in  Table  3.2-2. 


TABLE  3.2-2 

"SECOND-BEST"  STRATEGIES 


PHASE  1 

PHASE  2 

PHASE  3 

CD  Single  Industrial 

Incremental  -  Single  Source 

Licensing 

CD  Multiple  Industrial 

Incremental  -  Single  Source 

Single  Source  -  No  Options 

CD  Multiple  Industrial 

Incremental  -  Single  Source 

Licensing 

CD  Multiple  Industrial 

Incremental  -  Single  Source 

Leader/Fol lower 

CD  Multiple  Industrial 

Inereaental  -  Single  Source 

Second  Sourcing 

CD  Multiple  Industrial 

Incremental  -  Multiple  Sources 

Licensing 

Prototype  Single  Industrial 

Full  Concurrency  -  Single  Source 

Leader/Fol lower 

Prototype  Single  Industrial 

Full  Concurrency  -  Single  Source 

Second  Sourcing 

Prototype  Multiple  Industrial 

Full  Concurrency  -  Single  Source 

Single  Source  -  No  Options 

Prototype  Multiple  Industrial 

Full  Concurrency  -  Single  Source 

Licensing 

Prototype  Multiple  Industrial 

Full  Concurrency  -  Single  Source 

Leader/Follower 

Prototype  Multiple  Industrial 

Full  Concurrency  -  Single  Source 

Second  Sourcing 

CD  Single  Industrial 

Incremental  -  Single  Source 

Single  Source  •  No  Options 

CD  Single  Industrial 

Incremental  -  Single  Source 

Single  Source  -  Options 

CD  Single  Industrial 

Incremental  -  Single  Source 

Leader/Fol lower 

CD  Single  Industrial 

Incremental  -  Single  Source 

Second  Sourcing 

CD  Multiple  Industrial 

Incremental  -  Single  Source 

Single  Source  -  Options 

CD  Multiple  Industrial 

Incremental  -  Single  Source 

Single  Source  -  MYC 

CD  Multiple  Industrial 

Incremental  -  Multiple  Sources 

Single  Source  -  No  Options 

CD  Multiple  Industrial 

Incremental  -  Multiple  Sources 

Leader/Fol lower 

CD  Multiple  Industrial 

Incremental  -  Multiple  Sources 

Second  Sourcing 

Prototype  Single  Industrial 

Full  Concurrency  -  Single  Source 

Single  Source  -  No  Options 

Prototype  Single  Industrial 

Full  Concurrency  -  Single  Source 

Single  Source  -  Options 

Prototype  Single  Industrial 

Full  Concurrency  *  Single  Source 

Licensing 

Prototype  Multiple  Industrial 

Full  Concurrency  -  Single  Source 

Single  Source  •  Options 

CD  Single  Industrial 

Incremental  -  Single  Source 

Single  Source  -  MYC 
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3.2.3  Results 


A  summary  of  the  four  strategies  remaining  are  dis¬ 
played  in  Figure  3.2-3.  Note  that  if  a  more  stringent  com¬ 
parison  criteria  is  used,  only  two  strategies  remain,  2  and 
3  .* 


•  Multiple  source  contract  definition  followed 
by  multiple  source  incremental  FSD  followed 
by  a  single  source  multi-year  production 
contract 

•  Single  source  prototype  followed  by  single 
source  FSD  with  full  concurrency  followed 
by  a  single  source  multi-year  production 
contract . 


The  tradeoffs  between  the  two  options  are  as  follows: 


•  The  first  option  has  the  larger  proba¬ 
bility  of  success  for  the  earliest 
desired  IOC  and  the  lower  total  program 
cost 

•  The  second  option  has  the  larger  proba¬ 
bility  of  success  for  the  later  IOC  and 
the  lower  development  cost. 


*  As  discussed  in  Section  2.5.3,  the  dominance  analysis  portion 
of  the  model  incorporates  the  concept  of  "closeness."  One 
strategy  is  not  allowed  to  dominate  another  if  their  respec¬ 
tive  attributes  are  relatively  close.  This  is  accomplished 
by  comparing  attributes  against  a  pre-specified  threshold. 
Three  different  thresholds  are  used;  the  smaller  the  thres¬ 
hold,  the  more  stringent  the  comparison  criteria.  A  message 
is  printed  when  the  use  of  alternative  thresholds  varies  the 
results . 
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The  actual  strategy  pursued  by  Capt.  Hullander  for 
the  AN/SLQ-32  was  the  first  of  the  two  preferred  (CD  +  Incre¬ 
mental  +  MYC)  and  IOC  was  slightly  more  than  6  years  following 
program  inception. 

At  the  conclusion  of  this  exercise,  Capt.  Hullander 
remarked  that  he  and  his  staff  had  manually  performed  a  similar 
analysis  at  the  start  of  the  program;  they  itemized  what  they 
considered  to  be  the  candidate  alternatives  with  the  highest 
probability  of  satisfying  their  goals  and  sequentially  elimi¬ 
nated  "second-best"  alternates  based  upon  a  number  of  criteria. 

By  relying  principally  on  their  experience  and  judge¬ 
ment,  Capt.  Hullander  and  his  staff  spent  approximately  one 
month  in  the  early  days  of  the  program  accomplishing  this  ana¬ 
lysis.  Key  influencing  factors  were  as  follows: 

•  Moderate  technology  advance  combined 
with  compressed  schedule  requirements 

•  Small  inventory  requirements 

•  No  significant  constraint  on  R&D  money 

•  Ceiling  on  production  money. 

These  factors  were  applied  to  the  strategies  identified  as 
having  potential,  and  rough  estimates  for  both  time  and  cost 
were  generated  for  each.  The  chosen  strategy  emerged  from 
this  analysis. 

A  computerized  model  will  probably  never  duplicate 
the  judgement  and  insight  possessed  by  an  experienced  and 
intelligent  program  manager.  However,  the  degree  of  similar¬ 
ity  in  the  results  gives  credence  to  the  idea  that  a  fully 
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developed  model  properly  supported  by  data  might  provide  use¬ 
ful  assistance.  In  this  example,  perhaps  detailed  management 
analysis  could  have  been  applied  only  to  the  two  or  three  pre¬ 
ferred  strategies  output  by  the  model.  In  other  cases,  stra¬ 
tegy  alternatives  overlooked  by  the  staff  may  offer  potential. 
In  any  case,  supplementary  analysis  which  can  be  prov.ded  in  a 
short  time  period  (no  more  than  a  few  hours  including  sensi¬ 
tivity  analysis)  appears  justified. 
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4.  CONCLUSIONS  AND  RECOMMENDATIONS 


4.1  CONCLUSIONS 

The  examples  analyzed  in  Chapter  3  provide  ample 
evidence  that  the  prototype  model  provides  results  consistent 
with  program  experience.  The  examples  of  sensitivity  analyses 
presented  in  Section  3.1.3  through  3.1.5  illustrate  the  insights 
the  model  is  capable  of  providing  regarding  the  interrelation¬ 
ships  among  acquisition  strategy  alternatives  and  key  influ¬ 
encing  factors  such  as  the  following: 

•  Perception  of  technical  risk 

•  Urgency  of  need 

•  Development  cost 

•  Production  cost 

•  Estimated  inventory  requirement. 

The  concept  that  acquisition  strategy  en  mpasses  the 
entire  acquisition  process  is  emphasized.  A  uset  not  experi¬ 
enced  in  program  management  can  experience  first  hand  why  the 
selection  of  an  acquisition  alternative  for  one  phase  should 
not  be  made  independently  of  other  phase  options.  Further¬ 
more,  insight  into  the  importance  of  risk  identification  and 
risk  management  early  in  the  program  is  provided. 

Collectively,  these  findings  provide  a  strorz  indi¬ 
cation  of  the  potential  utility  of  the  model  as  a  teaching  aid 
to  program  management  students  at  DSMC  and  elsewhere.  As 
evidenced  by  the  AN/SLQ-32  example,  ASCM  may  also  be  able  to 
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provide  eerly  planning  support  to  a  program  manager,  given 
sore  in-depth  data  collection  and  analysis  to  support  the 
myriad  of  internal  relationships. 


4.2  RECOMMENDATIONS 

Recommendations  stemming  from  this  development  effort 
fall  into  three  categories: 


• 

Model 

implementation 

• 

Model 

update  and  refinement 

• 

Model 

expansion 

4.2.1 

Model  Implementation 

ASCM  should  be  implemented  as  a  teaching  aide  in  the 
program  management  curriculum  at  DSMC.  Although  the  prototype 
model  has  demonstrated  its  potential  utility  as  a  teaching 
aid,  the  current  software  is  inadequate  for  that  purpose.  The 
objective  was  to  demonstrate  model  capability  and  utility,  not 
software  elegance.  The  software  is  not  "user-friendly,"  sen¬ 
sitivity  analysis  is  laborious,  and  software  documentation 
consists  of  no  more  than  a  listing  of  the  source  code. 

The  prototype  model  should  be  expanded  into  a  vali¬ 
dated  software  system  with  additional  capability  needed  for 
successful  operation  in  the  teaching  and  research  environ¬ 
ments  : 


•  The  interface  between  the  model  and  the 
user  should  be  expanded  to  provide  a 
high  degree  of  assistance  and  protection, 
and  to  make  it  user-friendly 
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•  The  software  should  be  structured  to 
facilitate  the  answering  of  "what  if" 
questions  via  easily  executed  sensitiv¬ 
ity  analyses 

•  The  entire  system  should  be  well 
documented. 

4.2.2  Model  Update  and  Refinement 

The  prototype  model  contains  several  parameters  and 
relationships  which  are  considered  preliminary  in  nature  due 
to  the  lack  of  complete  data. 

The  specific  data  on  37  programs  plus  the  ancillary 
data  proved  sufficient  lor  prototype  demonstration.  However, 
since  data  provide  the  principal  determinants  in  most  relation¬ 
ships,  an  expanded  data  base  would  provide  vastly  increased 
validity  and  realism  to  the  model.  Additional  data  collection 
and  analysis  of  tactical  missile  system  and  electronic  sub¬ 
system  development  and  production  phases  should  be  performed. 
The  effort  should  expand  the  existing  data  base  and  focus  on 
validating  and  refining  the  following  model  parameters  and 
relationships : 


•  Second  source  start-up  costs 

•  Interrelationships  between  savings  due 

to  competition  and  savings  due  to  multi¬ 
year  contracting 

•  Effect  of  dual  sourcing  FSD  on  production 
costs 

•  Multiple  source  development  costs 

•  Similarities  and  differences  between  new 

system  development  and  existing  system 
modification 
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•  Interrelationships  among  successive  phase 
alternatives 

•  Risk  reduction  relationships  among  Phase 
1  and  Phase  2  alternatives. 


The  prototype  model  uses  numerous  techniques  and  method¬ 
ologies  to  implement  the  overall  concept  of  successively  reducing 
the  number  of  strategy  alternatives  to  a  small  set  containing 
the  preferred  alternatives  for  a  particular  situation.  Two  of 
these  methodologies  are  worthy  of  further  research  and  analysis: 


•  The  methodology  used  for  calculating  the 
probability  of  success  for  a  given  strat¬ 
egy  incorporates  an  assumption  of  indepen¬ 
dence  among  the  constituent  probabilities 
which  is  not  entirely  valid.  Additional 
research  into  the  most  appropriate  formu¬ 
lation  for  incorporating  identifiable 
dependencies  would  greatly  enhance  the 
validity  of  the  model 

•  The  dominance  analyses  performed  by  the 
model  incorporates  a  concept  of  "close¬ 
ness".  This  concept  implies  that  due  to 
the  softness  in  the  calculation  of  the 
attributes,  one  strategy  alternative 
should  not  dominate  another  if  their 
attributes  are  reasonably  close  to  each 
other.  This  concept  was  incorporated 
into  the  prototype  model  with  an  experi¬ 
mental  methodology  which  should  be 
refined  and  its  implications  analyzed. 


TASC  recommends  that  the  necessary  research  and  analysis  be 
performed  to  refine,  update,  modify,  and  validate  (as  required) 
these  two  methodologies. 


4.2.3  Model  Expansion 

In  order  to  provide  maximum  utility  to  the  defense 
community,  the  scope  of  the  model  should  be  expanded  to  include 
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other  categories  of  weapon  systems.  Detailed  data  collection 
and  analysis  for  additional  categories  of  weapon  systems  should 
be  performed  and  the  results  incorporated  into  the  model. 
Potential  candidates  include  the  following: 

•  Tracked  vehicle  systems 

•  Attack  helicopters 

•  Transport  helicopters 

•  Aircraft  (fighters,  bombers,  etc.) 

•  Ships 

•  Guns 

•  Specialized  electronics. 

The  scope  of  the  model  should  also  be  expanded  to  in¬ 
clude  a  model  of  the  concept  exploration  process.  The  primary 
objective  of  concept  exploration  is  to  examine  feasible  solu¬ 
tions  to  a  perceived  operational  need  and  to  select  for  further 
development  those  solutions  which  exhibit  the  highest  potential 
The  existing  model  begins  with  the  development  options  for  a 
pre-defined  concept.  Given  a  model  of  the  concept  exploration 
process,  a  student  could  begin  with  a  perceived  operational 
need  and  determine  likely  concepts  to  address  that  need.  Each 
of  these  concepts  could  then  be  processed  by  ASCM  and  the  trade 
offs  assessed  among  the  competing  concepts.  A  preliminary  in¬ 
vestigation  of  feasibility  is  recommended. 
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